Support catalog for Spark Snowflake #432
Conversation
zhaohehuhu
requested review from
sfc-gh-sshankar,
sfc-gh-ema,
sfc-gh-mrui and
sfc-gh-bli
as code owners
There was a problem hiding this comment.
@luoyedeyi Thanks for your contribution on the spark connector. Left some comments for this PR. Could you please take a look?
.github/workflows/snyk-issue.yml
Outdated
| @@ -0,0 +1,29 @@ | |||
| name: Snyk Issue | |||
There was a problem hiding this comment.
This file sounds not related to this PR.
.github/workflows/snyk-pr.yml
Outdated
| @@ -0,0 +1,31 @@ | |||
| name: snyk-pr | |||
There was a problem hiding this comment.
This file sounds not related to this PR.
| import scala.collection.convert.ImplicitConversions.`map AsScala` | ||
| import scala.collection.mutable.ArrayBuilder | ||
|
|
||
| class SfCatalog extends TableCatalog with Logging with SupportsNamespaces{ |
There was a problem hiding this comment.
Could you please check the scala style with sbt scalastyle?
| import scala.collection.convert.ImplicitConversions.`map AsScala` | ||
| import scala.collection.mutable.ArrayBuilder | ||
|
|
||
| class SfCatalog extends TableCatalog with Logging with SupportsNamespaces{ |
There was a problem hiding this comment.
It looks SfCatlog will work as a utility class for Spark Connector user correct?
Can you add some unit test and integration test for it.
| ident: Identifier, | ||
| schema: StructType, | ||
| partitions: Array[Transform], | ||
| properties: java.util.Map[String, String]): Table = ??? |
There was a problem hiding this comment.
??? sounds misleading. If you think the interface functions are not applicable for Spark Connector, I just you throw an UnsupportedOperationException
This comment applied to the other similar cases.
| import org.apache.spark.sql.types.StructType | ||
| import org.apache.spark.sql.{Row, SQLContext} | ||
|
|
||
| case class SfScan( |
There was a problem hiding this comment.
Can you explain how SfScan/SfScanBuilder/SfWriteBuilder/SfTable are used.
They are defined in this PR but I don't understand when and how they are used?
Could you please add some test cases for them?
There was a problem hiding this comment.
Sure. Just wanna implement some datasource interfaces already inside Spark to enable reading and sinking data on Snowflake by catalog.
| writer.save(data.sqlContext, data, saveMode, params) | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
Add empty line in the end of the file.
� Conflicts: � .github/workflows/snyk-issue.yml � .github/workflows/snyk-pr.yml
zhaohehuhu
force-pushed
the
Feature-428
branch
from
1cd52d0
to
be8d050
Compare
|
I refactored the code and please review it @sfc-gh-mrui |
No description provided.